Performance Confidence Estimation for Automatic Summarization

We address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either singleor multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for singleand 74% for multi-document summarization. Disciplines Computer Sciences Comments Louis, A. & Nenkova, A., Performance Confidence Estimation for Automatic Summarization, 12th Conference of the European Chapter of the Association for the Computational Linguistics, March-April 2009, doi: anthology/ E09-1062 This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/cis_papers/724